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§1.  Introduction. 

Assume  that  one  has  a  machine  whose  failure  is  described  by  a  wear  process 
x^it^O  which  is  a  positi  a  non-decreasing  Markov  process  and  a  random  threshold 
Y^O  independent  of  x^  :  t  ^  0  with  failure  occuring  at  time  a  =  inf  {t :  >  Y } . 

At  any  time  one  can  replace  the  machine  by  a  new  one  with  the  same  mode  of  opera¬ 
tion.  There  is  an  operating  cost  f(x)  per  unit  of  time  and  a  replacement  cost 
g(x)  if  replacement  is  done  before  failure  and  replacement  cost  c^  if  replace¬ 
ment  is  done  at  failure.  Note  that  replacement  is  always  to  replace  the  machine 
by  a  new  one.  Models  of  this  type  have  been  considered  in  the  reliability 
literature  by  Abde 1-Hameed  [1],  [2],  Drosen  [5],  and  Taylor  [10]. 

The  stochastic  control  problem  of  minimizing  the  cost  is  generally  called 
the  Optimal  Replacement  Problem  and  has  been  considered  by  the  above  authors  in 
the  case  that  f  is  constant.  The  interest  has  been  in  the  long  run  average 
cost  problem.  In  this  work  we  will  considers the  case  of  general  f(x).  We 
first  view  the  problem  as  a  discounted  cost  problem,  and  as  in  Robin  [8],  we 
obtain  the  long  run  average  cost  by  letting  the  discount  factor  go  to  zero. 

The  main  difference  with  the  work  of  Robin  is  that  the  invariant  measure  is  not 
obtianed  exponentially  fast  by  the  Markov  transition  probabilities  of  the 
replacement  process  as  time  goes  to  infinity.  (See  §4). 

In  Section  2  we  state  the  Long  Run  Average  Cost  Optimal  Replacement  Problem 
and  deal  with  a  preliminary  discounted  optimal  stopping  problem.  Section  3 
formulates  and  solves  the  discounted  optimal  replacement  problem.  The  replacement 
process  is  introduced  in  Section  4,  the  invariant  measure  is  found,  and  ergodic 
results  are  derived  for  the  linear  problem.  Section  5  proves  the  main  technical 
result  and  Section  6  contains  the  main  result  that  the  solution  to  the  discounted 
problem  suitably  modified  converges  to  the  solution  of  the  long  run  average 
problem. 
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Let  ft  =  D(R  ,  R  )  be  the  space  of  right  continuous  functions  with  left 

limits.  Here  R+  =  [0,  <*>) .  Let  x  (cj)  =  w(t)  for  uefi,  F°  =  o(x  :0<s<t), 

t  t  s 

F  =F  ,  and  F,  F  the  universally  completed  o-fields  F^  and  F^ 

00  t  t 

respectively.  Let  (ft,  F  ,  x^it^O,  P  )  be  a  homogeneous,  non-decreasing,  non¬ 
negative  Markov  process  with  associated  semi-group  T  :t^0  defined  on 
C^(R+),  the  space  of  bounded  real  valued  continuous  functions  defined  on  R+ 
with  norm  taken  to  be  supremum  norm.  We  assume 

(2.1)  T  :t^0  is  Feller,  that  is,  for  f  e  C,  (R+) 

t  b 

T^f  eC^(R+)  and  I  f  +  f  in  supremum  norm  as  t-+0. 

Let  A  denote  the  infinitesimal  generator  of  T  :t^0  and  its  domain  in 

C,  (R+) .  Assume  also 
b 

(2.2)  x  :t^0  is  quasi-left  continuous,  that  is  if  t  :n>l  is  a 

t  n 

sequence  of  stopping  times  with  t  +  t,  then  x  x  a.s.  P 

n 

on  the  set  (x  <  °°)  .  See  Dynkin  [6],  Vol.  1  pp.  103. 

Let  Y  be  a  positive  random  variable  independent  of  x  :t^0  with  a  continuous 
distribution  function  G(y).  Let  G(y)=l-G(v).  Define 

o  =  int{ t :x  > Y' } 

and  let  H(t)  =  P  (o<t)  =  P^(x  >Y).  Assume 

(2.3)  E  [o]  <  »  and  H(0)  =  0. 

Let 

(2.4)  f,gcC.  (R+)  f,g^0  and  c  >0  a  constant. 

b  o 

We  say  x:ft->[0,°°]  is  a  stopping  time  with  respect  to  x  :t^0  if  (x^t)eF 
for  all  t. 


A-V. 
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Let 


X  =  inf  En 
t  u 


/•a  AT 

/  £<XS)dS  +  I(T<<J)8(Xt)+I(T>a)C0 

J  n 


L  0 


Eq [ OAT ] 


The  Long  Run  Average  Optimal  Replacement  Problem  is  to  find  t  so  that 


Q 

> 

H  > 

o 

w 

II 

I  f ( x  )ds  +  I.-  Ng  (x-)  +  I, - ^  ,c 

1  s  (t  <  a)  t  (t  ^  a)  0 

-J0 

Eq[oAt] 


Our  first  step  is  to  establish  a  result  for  a  discounted  optimal  stopping 
problem.  For  the  following  see  Robin  [9]  or  Bensoussan  [3]: 


Lemma  2.1, 


For  b^O  fixed  and  a>0,  let  V,  e  C.  (R  )  be  the  maximal  solution 

b  b 


of 


U  c  cb<R  )  U(x)  <  b  +  g(x)  G(x)  +  CqG(x) 


rt  u+f 

J0 


U^e  atT  U  +  |  e"018!  (fG  +  a(b  +  c  )G)ds 

s  o 


then 


V,  (x)  =  inf  J  (t) 
b  x 


where 


e  °S(f(x  )G(x  )  +  a(b  +  c  )G(x  ))ds 
s  s  Us 


+  e  aT(b  +  g(x 


t)G(xt)  +  coG(xx)) 


Moreover  if 


t,  =  inf  { t  :V  (x  )  =  b  +  g(x  )G(x  )  +  c  G(x  )}. 
b  b  t  t  t  U  t 


then 


Vx)  =  Jx(V‘ 


Since  H  is  not  point  mass  at  0,  H(a)  <  1, 


‘•If 


e  “S(f (x  )G(x  )+a  cA  G(x  ))ds 
s  s  0  s 


and  we  have 


Vb(0)  <zQ  +  b  H(a) . 


Select  t>g  large  enough  so  that 


W(“)<bo 


Next  for  such  a  b^,  we  have 


bl  ■  V0,<bo 


and  by  Lemma  2.2,  it  follows  inductively  that 


b,.  =  (0)  <VL  (0)  =  b, 


k  b 


Define  b  =  limb^,  and  let  Vg(x)  be  given  by  Lemma  2.1, 


From  Lemma  2.2, 


lvb  <*)  _vb(x)  I  <2lbk“bl 

k 


and  so  V,  +  V-  in  C,  (R  )  and  moreover 
b.  b  b 

k 


b  =  lira  b  =  lim  V.  (0)=VC(0). 
k  k  k  bk-l  b 


We  thus  have  established  the  following. 


Theorem  2.4.  Under  the  assumption  (2.1)  -  (2.4),  let  V  (x)=V-(x),  b  as 


in  Lemma  2.3.  Then 


Va(x)  =  inf  J  a(t) 

T  X 


where 


J>)=E 


.[/• 


aS(f(x  )G(x  )  +a(Va(0)  +cn)G(x  ))ds 
s  s  Os 


+  e  aT(Va(0)  +g(xT)G(xT)  +cqG(xt)) 
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Moreover  if 

T =  inf{t:Va(xt)  = Va(0) + g (x  )G (x  )  + cQ  G(x  )  J 

then 

V  (x)  =  J  (t) . 

X 

Remark  2.1.  It  will  be  shown  in  Section  3  that  Va(x)  is  the  maximal  solution 
of  a  quasi-variational  inequality. 

Corollary  2.5.  Under  the  assumption  g(0)  >0  and  these  of  Theorem  2.4, 
x>0  a .  s .  PQ 

Proof .  If  g(0)  >  0  then  taking  t  =  0 
Va(0)  <  Va(0)  +  g(0) 

and  hence  there  is  a  6>0  so  that  if  0^x<6  then 
Va(x)  +  e<v“(0)  +  g(x)G(x)  +  cQG(x) 
for  e>0  sufficiently  small.  Hence 

t  >  t ^ ^  =  inf  (t :x(t) > 6/2  ) 

and  since  x  :t>0  is  right  continuous,  t.,„>0  a.s.  P_. 

t  o/2  U 


§3 .  The  Discounted  Optimal  Replacement  Problem, 

Let  x^it^O  and  Y  k^l  be  independent  copies  of  x^:t^0  and  Y. 


Def ine 


a  =  inf  Yk| 


Let  f,  g  and  satisfy  (2.4).  Suppose  T^rk^l  are  stopping  times 

fk  k 

respectively  with  respect  to  F  :t^0  the  universal  completion  of  oCx^is^t) 


and  assume  for  all  n 


t  =  co  a.s.  P  .  We  use  the  notation 
k  o 


t=(Tj,  and  refer  to  T  as  replacement  times. 


Define 


r  rvh 

Jo(T)  =  Eo  I  e'aSf(xi)di 
L'o 

00  n  -  1  at  /  n 

+  E  n  e-aaAT£( 
n  =  2  £  =  1  VL 


hds  +  e  aolATl  (g(x^  )I.  \  +  c  1/  > 

s  V  (Tl<al>  °  (T1>0 


“Sf(xn)ds 

s 


+  e  a0nATn(g(xn  )t  +  c  t  ) 

Tn  (Tn<0n}  °  (tn >  °r?  I  ' 


Here  it  is  assumed  that  Pq(Xq  =  0)  =  *  for  all  k.  Let 


,.a  .  -ra  , -a. 

V  =  mt  J  (  t) 
o  — *  o 

T 


We  seek  T  so  that 


V®-  J“(t) 
o  o 


This  we  refer  to  as  the  Discounted  Optimal  Replacement  Problem. 
Define 

r  rVT i 

,  /  v  r.  I  i  -as „  .  lx  ,  .  -aaiAx,  ,„a  .  1 


r  r  i  Ti 

Tl)=Eo  I  e'aSf(x])ds  +  e-a°lATl(Vao+g(xMI(  <a  ) 
~Jr\  III 


+  c  I, 
O  ( 


W>_ 


I 

sfl 


Lemma  3.1.  V  =  inf  J  (t.) 
-  o  o  1 

Ti 


Proof .  Let  t  =  (x^ , , . . . )  be  any  replacement  sequence. 
By  independence  letting  x =  (t^ ,x^ , . . . ) 


Ja(  x  )  =  E 
o  i 


Therefore 


+  c  I . 
o  (x 


v“>inf  Jjd  jid 

T1 


r  r  i  i 

:o  e“aSf(x1s)ds  +  e-a°lATl(j“(^)+g(x1  )I(  } 

LJ0  111 

+  CoI(x1  >  c^))  >  Jp (t i ) 


>  r(x.) 

o  1 


For  e  >  0  choose  x^  and  x  so  that 


j”(x1)<d  +  e  and  (  x" )  <  v“  +  e 


Let  x„  = 


(XjiX)  then 


Vo^o<0<Jo(V  +  e<d  +  2e' 


Since  c  was  arbitrary,  we  have  equality. 

In  what  follow  x^:t>0  and  Y  wil  be  generic  copies  of  the  wear  process 
and  the  random  threshold  and  a  =  inf { t  :x^_  ^  Y}  .  Note  (2.3)  insures  o  <  03 
a.s.  Pp. 

Lemma  3.2.  for  any  stopping  time  x  with  respect  to  F  :t2*0  on  the  set 


(x  <  00 ) 


Po(o^x|Ft)  =  Pq(xt^y|Ft)  =  G(Xt) 


P  (a>x|F  )  =p  (x  <  Y I F)  =G(x) 
o  1  x  ox'  x 


On  the  set  (x  =■  <*>)  , 


P  (a^x|F  )  =  1  and  P  (o>x!F  )  =  0. 
o  x  o  ■  x 


FSW* 


y  / 


l-v 

tv 

fiv 


Ay 

,V 

\S 

ft 


V 


K-: 


g 


Proof .  First  note  that  by  the  independence  of  x^it^O  and  Y,  and  the 


assumption  that  x  :t>0  is  pathwise  non-decreasing,  for  any  fixed  t. 


P0  ( o  <  1 1  F ^ )  =  PQ  (x^_  >  Y  |  )  =  G(xt) 


For 


any  t  an  F^tt^O  stopping  time,  define 


'""1 


k/n  on  (k-l/n  <  i  ^  k/n)  =  A^n 
00  on  (t  = «) 


Since  A^n  e  ^/n>  on  the  set  ^Tn  < 

00 

Pn(o<T  I F  )  =  V  P  (o^k/njF.  ,  )I 
°  n  t„  k/n  \/„ 


oo 

Eg  (x  >i  -  g(x  > 
k/n  \/„  Tn 


Because  t<t  ,  F  <F  and  so  on  the  set  (r<°5) 
n  t  t 

n 


P0(a<TlFT)  =  PG(  *  (0<Tn)iFT) 


=  lim  E0(P0(a<Tn|FT  )  |  ] 


=  lim  E  [G(x  ) ! F  ]  =  G(x  ) 


The  last  follows  since  t  it  and  x  :t^0  is  right  continuous. 

n  t  & 

On  the  set  (t  =  °°)  there  is  nothing  to  show. 


Lemma  3.3.  Let  f  and  g  be  bounded  and  continuous  and  a>0.  Then  for  am- 


stopping  time  t  with  respect  to  F^:t^0. 


(i) 


[/. 


a  At 


e'aSf(x  )ds 
s 


■]  ■  ^  [f 


-as , 


e  f(x  )G(x  )ds 
s  s 


-V  -~A  -  .  _  A  .V  -  J.  V  a  Cj.  L  A  P.  v 


(ili)  E0[.-"A7I<t><,)]  -  E0  [  e'“,G(xi)  +  T 

~f\ 


ae_ctSG(x  )d 
s 


Proof.  (i)  By  Lemma  3.2 


P  (x  <  Y  F  )  =  G(x  ) 
os  s  s 


and  so  on  the  set  (s<t) 


P  (x  <  Y  F  )  =  G(x  ) 

O  S  1  i'  g 


Hence 


r 


e  aSf(x  )ds 
s 


]  ■  •■[£ 
tf 


e~CtSf  (x  )I 

s  (x 


E^j  |  e  aSf(x  )G(x  )ds 
s  s 


3 


(ii)  This  follows  directly  from  Lemma  3.2, 
(Hi)  Note  that 


—no At t  -ax  _ 

e  I/V>.=e  I, 

(t^o)  (x 


~  ~aST  ,  T 

ae  I.  s v,ds  I, 
(x  >Y)  (x 
s 


Since  x^:t^0  is  pathwise  non-decreasing 


“““  X(X  >Y)ds  1 
s 


rx 

I  -asT 

(x  >Y)  J ae  (x  ^ 

T  S 


Y) 


ds 


The  rest  is  Lemma  3.2. 
Lemma  3.4.  V 


*  •  1;f  Eo[/ 


e  aS(f (x  )G(x  )+a(Va  +  c  )G(x  ))ds 
s  s  o  o  s 


+  e 


-ax 


<v“ +  g(x^)G(x^)  +  coG(xt>)J 


Proof.  This  is  immediate  from  Lemma  3.1  and  Lemma  3.3. 


Theorem  3.5.  Suppose  g(o)  > 0  and  (2.1)  —  (2.4)  are  satisfied. 
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the  maximal  solution  of  U(x)^U(0) 

(3.1)  UeCb(R+)  U(x)<U(0)  +  g(x)G(x)+c0G(x)  and 


U(x) <  e"atTtU(x)  + 


/*te"aSTs[fG  +  a 
•'O 


(U (0) +  Cq)G] (x)ds 


then 


(3.2)  Va(x)  -  inf  Ja(T)  and 

T  x 


with 


T *  inf }t:Va(xt)  *  Va(0)  +  g(xt)G(xt) +  cQG(xt)| 

(3.3)  Va(x)  *  j“(x) 


Moreover 

(3.4)  Va(0)  -  inf  t  ) 

T 

* 

*  »  /s 

and  if  x**(Xj,  where  x^  is  defined  the  same  as  x  above  except  xt 

replaces  Xf,  then 

(3.5)  Va(0)-J®(x). 

Proof.  Let  Va(x)  be  as  in  Theorem  2.4.  By  Lemma  2.1,  Va(x)  solves  (3.1). 
Moreover  (3.2)  and  (3.3)  are  consequences  of  Theorem  2.4.  Also  Lemmas  3.1  and 
3.4  prove  (3.4).  What  is  left  is  to  show  (3.5)  and  VH(x)  is  the  maximal 
solution  of  (3.1). 

Let  U(x)  b.  any  solution  of  (3.1)  and  x  be  any  stopping  time.  By  the 
Markov  property. 


-at  , 
e  U(x 


r 


e"aSf (x  )G(x  ) +  a(u(0) +  c  G(x  ) )ds 
s  s  OS 


is  a  submartingale-  Hence 


+  a(U(0) 


e  aS(f(x  )G(x  ) 
s  s 


+  c  )G(x  ))ds 
o  s  J 
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Bv  (2.2)  letting  t 


(3.6)  U(x)<E 


Je  aTU(x  )  +  f 
•*0 

■11} 


e  °S(f(x  )G(x  )  +  a(U(0)  +  c  )G(x  ))ds 
s  s  os 


] 


I  /  e  aSf(x  )G(x  )  +  a(U(0)  +  c  )G(x  )ds 
“  s  s  OS 


+ e~aT(U(0) 


+  g(xT)G(xT)  +  coG(xx))j 


If  we  cah  show  U(0)  <  V  (0)  then 


U(x)<JX(t)  and  so  U(x)^V°(x) 
a 


Bv  (3.6)  and  Lemma  3.3 


OAT 


(3.7)  UCOXlJJ  e-“sf<xs)dS  +  e-aoA1(0(0)  +  g(xt)I(t<o)  +  coI(T>o)) 


'] 


If  t *  <Tj ,t2, . . . )  is  any  replacement  sequence,  then  using  (3.7)  inductively, 
after  n  steps  we  have 


(3.8)  U(0)^Eo  f  e"aSf(xl)ds  +  e'aaATl  (g(xj  )I(t  0  }+  Vor.^o.)5 

-  •'n  ill  II 


n  j  -  1 


-VTJ 


e  aSf(x'J)ds 
s 


f  L  n  e^VW  f 
j-2  £-1  JQ 

1 

-ao,  AT,,  /  3  ,  t  n  -ao,  At»„,  . 

+  e  j  :<g<XT  >I(  )+c  I  e  t  tv( 0)1 

S3!  j  J  l "  1  J 

By  (2.3)  since  the  o^'s  are  independent,  and  = 00  for  all  n  a.s.  Pq, 


j  >  n 


we  obtain  by  letting  n->® 


U(0)<J  (t),  i-e.  U(0)<V  »V  (0). 
o  o 
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\r» 

ft 


P/.v 


'  s 


ft 


ivp' 

l»Y) 


iw 


ft? 

& 


S 


y. 

M  * 


To  obtain  (3.5),  repeat  the  above  argument  using  Va(0)  and  t.  At  each 
step  we  have  equality  and  so  (3.8)  is  an  equality.  Corollary  2.5  assures  that 


5V 


a.s.  PQ,  and  so  Va(0)  «  J^(t), 


v.-- 


1A 


§4.  The  Replacement  Process. 


If  lr  Jq  ^ 

Let  x  :t>0  and  Y  be  as  in  Section  3  and  as  before  o,  *  inf  { t  :x  ^  Y  }. 

t  iC  t 


Assume  that  Pq(Xq-0  for  all  k)  -  1  and  define  0^  =  0.  Let 


(4.1) 


zt«xt  on  the  set  ^  <  t  <  ^ 


k  -  1 
0 


tm  0 


We  refer  to  zt:t^0  as  the  replacement  process.  Let  H(t)  be  as  in  (2.3). 


For  BCR  ,  Borel,  by  Lemma  3.2  and  the  independence  of  x^st^O  and  Y  , 


l(zt£B)'  LP0(  Zo^t,  \-k-l  eB*  Xt-k-l<Yk) 

k-1  '£-0  T]  op  V  On  / 


L  ° l 
1-0 


Z  «  0 


-  E  /  EolVXt-u)G(\-u)]  H 

k-1  J0 


(k-1) 


(du) 


a 


Pt  _ u(0,dz)G(z)R(du) 


where  (t)  is  the  k-fold  convolution  of  H(t),  R(t)  -  ^H'^'Xt),  and 

0 


,00, 


Pt(f),dz)  -  Pq(x^ e dz).  It  is  standard  from  renewal  theory  (see  Feller  [7]), 


that  R(t)  < 

Suppose  G(x)  <  1.  Considerations  of  x  for  which  G(x)  -  1  are  not 


necessary  because  z^ztX)  never  reaches  x.  Define  by  defining  f  r 


y  >  x 


P(Y  <  v)  -P(Y<y|Y  >  x)  ■=  -  (.XlzSJ.x)  =  G  (y) 
X  G(x)  X 


We  use  the  notation  P  .  to  stand  for  the  condition 

x,0 


lk  I 

Px  0(^-x,  x^-0,  k^2)  -  1.  Take  Y^  to  have  distribution  G^(y)  and 
tain  the  usual  independence. 


nain- 


Define 


o  -  inf{t:x*  ^Y1}. 
x  t  x 

Let  z^:t^O  be  as  in  .  (4.1)  except  that  0i“0x* 

Define  H^(t)  * < t) .  By  a  slight  modification  of  Lemma  3.2, 
(4.2)  Vlt£B)-Px,XEB,  «Uy‘) 


£  Px,o(  L  X^-k-l  EB'x^-k-l  - 

2  l*  0 

£» 0  £« 0 

JVVVV'-  £  [  VW^Vu^x*" 

k*  2*'0 


/pt(x.dy)^l+j  jp  ( 0,dy)G(y)H 

•4  e<*)  Jn  -4 


*R(du). 


where  Pt<x,dy)  -  e  dy) . 

To  establish  the  Markov  property,  first  note  that 

(4.3)  px^^-xs<Yx^s  +  t)^x1IA(xs)(Gx(xS+t)-Gx(xS))1 


G(x  ) 

E  [I(x  )  - G  (x  )] 

x  A  s  — ,  .  xo  s  + 1 
G(x)  s 


/ Ps<*' 


dy)  H  (t). 

G(x)  y 


zt  e  dy)  =  (0 ,dy)  *  Pt_u(0,dv)G(y)R(du) 


G(y) 

Px(Ztedy)  =  Pt(x,dy)  -(x)  + 


j  Pt_u(0,dy)G(v)Hx*R(du) 


The  Markov  property  for  zt:t^0  will  follow  if  we  establish 
Px(zsEA,Zs+teB)  ■/  / qs(x,dy)qt(y,dz)  . 

A  *4 


(4.4)  P  (x1  e  A.x1  <  Y^x1  ,  „  c  B,  *<yS 
xs  s  xs+t  s+t  x 


“  IP  (x,dy)  IP  (y,dz)  G~— - 
A  B 

-  /p  (x, dy)  3^-  /p  ( y,dz) 

*'  G(x)  •/  G(x) 


5y  (4.3)  we  have 


°°  i  J  _  2, 

(4-5)  r  px,o(xseA*xi<Yx*z  a£<s+t>^+t-j-i  £B*xl+t-j'1 

i  * 2  *-0  £  X) 

£«  0  £-0 


*H(j_2)(du) 


(4.6) 


V  /p  <x,dy)  ^  /  fp  (0,dz)G(z)H 

3-2  4  G<*>  Vb  ’ 

/P  (x.dy)  /  IP  (0,dz)G(z)H  *Rl 

S  g<x>Vb 


Lp*.o(  E1°i<s-1's-k-iEA-%-k-i<Yk-  V°t<s  +  t> 

k=2  £-0  S°£  £  °£  £-0 


£-0 


£-0 


x+t._k-l  e  B*x 


s  +  t  -  k  "  1 


'  E  |  f  Ps_u(0,dy)G(y)  y Pt(y, 
V  -  2  *'0  *4  B 

■f  A>g_u(0,dy)G(y)  ypt(y,dz)  3^ 
4)  *4  B  G(y 


£-0 


dz)  H  y'H(k_2)  1 

G(z)  X 


dz)  H  *R(du)  . 

G(y)  X 
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Again  by  (4.2) 
(4.7) 


k_1  k  k 

s  *xs  _  k  -  1  eA*xs_k-l  <\- 

k  -  2  j  -  k  +  1  '  £  «  0  £  °£  £  °£ 


OD  00  k  ~  1 

£  .£  px.o  £ 

,  »  o  _  i,  j.  i  \p«-n 


£  *  0 


£  •=  0 


j  -  1 


£  °£<S  +  t*Xs  +  t- 3  ~  1  £B,X 


s  +  t 


-j-i  <vj) 


£  =  0 


£-  0 


£  *  0 


£  I  [  /pa_u(°.dy)G(y)  j  Jpt_u(0,dz)G(z)Hv*H(j_k'1)(du) 
:  =*  2  j  *  k  +  1  **  0  A  *0  B  '  . 


k  -  2  j  ~  k  +  1  0  A 

*  s  j» 

(0,dy)G(y) 


H  *H(k_2)(du) 
x 


f  /*8-u(0*dy)G(y)  f  fpt_u(0* 

•'o  /a  *o  b 


dt)G(z)H  *R(du)H  *R(du) 

y  x 


Now  by  conbini-ng  (4.4)-(4.7), 

P  (z  e  A»2  e  B) 
x  s  t+s 


«  oo  /k  -  1 

£  £r*,o(£^ 

=  1  j  =  k  '£  =  0 


j  =  1  j 
k  -  1 


E,x  k  -  1  e  A* 


£  =  0 


_k  !  <Y  .  £  «’*<•  +  «:, EB.^+t 

£  £° 


2  °o  £*  o 


j-t 

r'  ' 


£  =  0  £-0  £  =  0 

JPc(x,dy)^i  Ip  (y,dz)£&-+  I  Ip  (0,dz)G(z)H  *R(du) 
/  G(x)  <  1  G  (x)  X  1  t_U  y 


0  3 


~f  I Pg_u  (0,dy)G(y)  j  Pt(y,dz)+  I  I  ,dz)G(z)H^*R(du)  Hx*R(du) 

*0  "A  ■'b  *0  B 

■//q  g(x,dy)qt(v ,dz) 


Next  let 


S  f(x)  =  E  [f  (z  )  ]  =  ± -  T 

C  X  G  (x) 


f  * 

(Gf ) (x) +  Tf  (G 

t  I  t-s 

*  A 


(Gf)(0)H  *R(ds) 
x 
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For  fe  C.  (F  ),  since  G(x)  is  assumed  continuous  and  T  :t>0  is  Feller 
b  t 


1 


—  T  (Gf ) (x)  e  C  (R  ) 

G(x)  C  '  b 


Moreover 


H^(t)  =  E^fG^CX^) ]  is  continuous  in  x  and  therefore  the  family 


of  distributors  {H  (t):xeR  }  is  continuous  in  distribution  in  x.  Since 


T  s(Gf)(0)  is  continuous  in  s  and  H^*R(t)  is  continuous  in  t, 


T  (GI)(0) H  *R(ds)  is  continuous  in  x  and  so  S  f e  C,  (R  ). 

t-s  x  t  b 


To  show  that  is  strongly  continuous  we  assume 


(4.8)  G  e  D  and  AG-— ■  is  continuous  and  bounded. 

Pi 


G(x) 


Note  that 
(4.9) 


iv,'0 


(G)(0)Hx*R(ds)  *  ExtGx(x  )]. 


so 


|s  f(x) -f(x)|  <  |T  f(x)  -  f(x)|  +2||f||T  G  (x) 
t  t  t  x 

From  (4.8)  by  Dynkins  formula 


°<  T  G  (x)  * 


G(x) 


(TtG(x)  -  G(x) )  = 


=*-/ 

G(x)  J. 


T  AG(x) 
s 


<  tl!  —  I! 


and  therefore 


|S  f  -  f  |U  |!t  f  -  f|!  +  2|1  f|l  l^llt 
C  C  G 


and  we  have  strong  continuity. 

To  comDute  the  infinitesimal  generator,  observe  that 


M(«)  .  Ii.  Stf(x)-£(x) 


t  ->0 


.  .  1  T  (Cf)(x)  -  G(x)f (x) 

lim  -  t  + 


t  -*■  0  G(:.) 


“o  UP'-“<0' 


dv)G(v) f (v)H  *R(du) 
x 
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Pointwise  then 


lim  1  Tt(Gf)(x)  -G(x)f(x) 
t  -*•  0  G(x)  t 


A(Gf)(x) 
G(x) 


and  »t  .  . t 

7  J  Pt_u(0,dy)G(y)f  (y)Hx*R(du)  -  pt_u(0,dy)G(y) (f  (y)- 

0  t) 

rl 

(4.10)  +  f(0)  lit  1  p  (0,dy)G(y)H  *R(du) . 

t-o  t  J0  c"u 

Clearly  the  first  term  on  the  right  of  (4.10)  goes  to  0  and  by  (4.9) 

limn  “  f  P  (0,dy)G(y)H  *R(du)  -  — — 

t  -*•  0  t  |  t-u  *  J  7  x  t-»-0  — .  . 

Jn  G(x ) 


f(0))H  *R(du) 


TtG(x)  -  G(x) 


AG(x) 

*  — — —  # 

G(x) 

Therefore  pointwise 

(4.11)  If <x)  .  +  f(0)  ^£CXJ>. 

G(x)  G(x) 

With  respect  to  suprewum  norm  on  C^(R+)  we  establish  the  following:  Under 

—  AGf 

the  condition  (4.8).  f  e  D-r  if  and  only  if  Gf  e  D  and  -  is  bounded  and 

A  g 

continuous.  We  first  note  that 


(4.12) 


7  I  P,  (0,dy)G(y)f(y)H  *R(du>  -f(0) 

C  Jn  t_U  G (x) 


<  Uf\ 

|  t  I  t-U 


(G(f  -  f (0) ) (0)H  *R(du) 


■rlf(0)l|f  Vu 

Jn 


G(0)Hx*R(du)  - 


AG  (x) 
G(x) 


From  (4.8),  (4.9)  and  Dynkin's  formula,  for  t  small  enough 


It/  WG(f- 


f(0))(0)Hv*R(du)jl<  sup  ||TcG(f  -  f  (0)  j 'I!  ^-H*R(t) 


0  ^  s  ^  t 


1  { G  (x  )  -  G  (x)  ] 


G(x) 


< 


o<s<t  ’’VG(f  - f(0))l!  c  P  f-o. 

G 


Moreover  by  (4.8),  (4.9)  and  Dynkin  s  formula  again. 


ifx 

tj  t-u 


(G) (0) H  *R(du)  - 
x 


AG  (x) 


=  TtG(x)-G(x) 
G(x)t 


AG  (x) 


if  ".(f)- 


^  1  ]  |  ds!' 


G  \  G  (x) 


Now  the  first  term  goes  to  0  and  for  the  second. 


|i  T  G  -  G  ,, 
s  lids 


°  G 


(4.13)  7  :  f  T  (  zr  (  — - iYU|M-»F  f  E  V-G  lids 

J0  SU  Vg(x)  G  j0  t  G 

<!^1!2  i  f\  ds 

G  o 

and  so  we  have  uniform  convergence  for  the  left  hand  side  of  (4.12). 

Next  if  f  e  D',  then  by  (4.11)  and  the  fact  that  we  have  uniform  conver 
A 


mce  in  (4.12) 


~  OF(Gf)  -  Gf)  -  AGf!1  <  l:Tt(Gf)  Gf  -  AGf 


t  t 


S  f  -  £ 


Af  1  +  |! 


if  h-u 


AG  (x) 

(Gf ) (0)  H  *R(du)  -  f (0)  . 

x  G  (x) 


linl1:-  (T  (Gf )  -  Gf)  -  AGf  II  =  0 
t  ->  0  t  t 


is  bounded  follows  from  (4.11). 


Converselv  if  Gf  e  D  and  is  bounded  to  show 

A  G 

!  t  -  Af  I-  -*■  0  we  must  show 


ii  T £  ( C  f )  -  C. f  A(Gf  )  n 


This  is  so  since 


VC’f)  “  Cf  -  A ( C f )  :  < 


+  r  -  I  f 
*  tj0 


if  ■'.(* ?k 


ds 


T  f  G  ^  “  C  ii 

s V  '  ,ds-*-0.  The*  latter  follows  as  in 


(4.13). 

To  summer ize,  we  have  established: 

Theorem  4.1.  Assume  the  conditions  (2.1)  and  (4.8).  Let 


z  =  x  t-  _  i  on  the  set 

t  t  -  K  1 

z  °c  f-o 

£-0 


k  -  1  k 

o „ <  t  <  y  c." 

f «  o 


E  wc 


Then  z^it^O  is  a  strong  Markov  process  and  has  transition  probabilities  given 


bv 


q  (x,dy)  =  p  (x,dy)  +  I  p  (0, 

C  *  G(x)  JQ  t_U 


dy)G(v ) H  *R(du ) 


Moreover  the  associated  semigroup  S  :t^0  acting  on  C^(R  )  i">  strongly 

continuous  and  Feller 

Further  letting  Af  denote  the  infinitesimal  generator  then  f  z  D-f  if 

and  onlv  if  Gf  cD  and  is  bounded.  Lastly 

A  G 

Afoo  -  MStHii  +  f(0)  . 

G(x)  C(x) 

Theorem  4.2.  Suppose  (2.1),  (4.13)  and 


14.14)  E  [a  ]  <  °°  for  all  x. 
x  x 

Then  r  :  t  0  has  a  unique  invariant  orobability  measure  fl  niven  b* 


C°  1 

_  ~\ 

eJ 

1  I.(xjdt 

E  f 

1  I  ( x  ) G ( x  )dt 

=  o[ 

Jo  A  c  J 

= 

;0  a  c  c  J 

E0!o] 


[/X-] 


Remark  4.1. 


In  the  work  of  Robin  [8]  it  is  required  that 


p  (z  t  n-ncn  <Be-Yt 

X  t 


>  0 


and  as  a  consequence 


Stg(x)  -  n(g)  ^B1e"",']t  Y>° 


vnere 


=  ./s(x) 


n< c, )  =  yg(x)  n  (dx) 

For  our  case  we  have  only  that 


lirr.  1  f  _  , 

t-0  I  /  SueU 


x)du  =  n(g). 


Since  z^rt^O  is  periodic,  we  will  not  even  have  pointwise  convergence 


Let  f  c  C,  (R  )  and  u  (x)  be  the  unique  solution  of  Au  -  au  =  -f 
b  a  a  a 


It  is  standard  that 


(4.15) 


Define 


Theorem  4.3.  Under  the  assumption  (2.1),  (4.8)  and  the  assumption 


r‘ 
-  Jr\ 


E  I  G(x  )ds 
X  I  s 


(4.16)  E  [c  ]  = 

X  X 


independent  of  x. 


C(x) 


(4.17) 


P  (X  <«)  =  !  for  all  t  and  x,  and 


lim  x^  =  °°  a.s.  P  for  all  x. 

t  -ao  C  X 


lim  au  (x)  =  f 
a  -+■  0 


(ii)  Let  v  (x)  =  u  (x)  -  a  (0).  Then 

a  a  a 


liTn  =  v(x)  uniformly  on  compact  sets  where 

a  0 


ExL)  (f  "  f)G(x 

u 


v(x)  = 


‘s)<is] 


and  v  is  the  unique  solution  of 


-Av  =  f  -  f  with  v(0) =  0. 


Proof.  From  (4.2)  and  (4.15) 


u  (x)  =  f-  E 
G(x)  X 


By  (4.16) 


_  00 

[/•- 


'f (xt)G(xt)dt 


i  e  r  f 

i  G(x)  X  L-J 


oa  £ 

fe-aC  f  E0[f<*t_s)G<*_.>]  VR(ds)dt- 
00 

CO 

00  E  I  G (x  )dt 

r  r  1,  x  LJ  c  J 


e  atf (xt)G(xt)dt  ^  f 


and  so 


i 

a  !  z  E  [  I  e_atf(x  )(T(x  )dt 

Glx)xL^  C  C  J 


Next  by  the  convolution  property  of  Laplace  transforms. 


“  *  t  \  ^  >  .  .  „■  ■  *  i<,  »  i  «  »  *  .  »  ,  -  ,  ►  *  f  s  {  j>  *  S  *«  , 


J  °l 

•'n 


x  )G(x  )  H  *R(ds)dt 
c  -  S  t  -  S  J  X 


e  atE  [G 


:Q[G(xt)f  (xfc)]dt  | 

Va 


e“atHx(dt)  I  e"CttR(dt) 


a 

Va)  =  J 


e  H  (dt)  and  note  that  H  (ot)  — *•  1  as  a-*0. 
x  x 


Since 


e  atR(dt)  = - - -  where  H(a)  =  I  e  atH(dt) 

l-H(a)  Jn 


we  have 


llmn  -(-a)  «  E  [o]  =  E  T  f  G(x  )dtl 

a->-0  a  o  °  l  J  c  I 


showing  (i). 


For  (ii),  note  that 


oo 

ua<0,"Eo  [/e"<’t^<xt>f<!!t)dt_  T^S 


H(a)  * 


Therefore 


(4.18) 


u  (x)  -  u  (0)  =  —  E  I  e  G (x  )  f  (x  )d t  j 
a  a  G(x)  X  L  J0  C  J 

r  r  i  h  (cx)-i 

+  E  I  c'at  G (x  )  f  (x  )dt  — -  . 

^  /  J  l-H(a) 


It  is  enough  to  establish  the  uniform  convergence  on  compact  set  for  f^O, 
the  general  case  easily  follows. 

CO  OO 

-  E  ["  f  e"at  G(x  )f  (x  )dtl  ♦-  E  f  f  G(x  )f  (x  )dt~] 


which  is  bounded  by  (4.16).  Next 


lim  Ell 

“*°  °U0 


e  at  G(xt)f (xt)dt 


1  -  H(a) 


=  f 


For  the  other  part  of  the  second  term  in  (4.18), 


H  (a)  -  1 
x 


[f 


EX  1  i  ae  at (G(xt)  -  G(x) )dt  -  G(x) 


) 


a  G(x) 


E .  |  |  e~at  G(xt)dt 


cc 

-U 


oo 

!'v" 


G  (;:) 


G(x) 


Therefore,  uniformly  on  compact  sets 


lim  u  (x)  -  u  (0) 
n  a.  a 

a  ->  0 


00 


f)dt 


G(x) 


To  show  that 


-A v (x)  =  f  (x)  -  f 


first  note  that  v(0)  =0  and  so  by  Theorem  4.1  it  is  enough  to  show 


-A(G  v ) ( x )  =  G(x) (f (x)  -  f). 


Note  that  by  the  Markov  property 
G(x)v (x) 


G(x  ) (f (x  )  -  f )dsl 
s  s  j 


G(x  ) (f (x  )  -  f )ds  + 

'0  S  S  ■'0 


f  cut 

•'n  L 


.  )(f(x_  ) 

+  s  t  +  s 


G(x  )(f(x  )  -  f )ds  +  G(x  ) v(x  ) 1 
s  s  t  t  J 


I 


And  so 


T  G  v  -  G  v 

r  t 


+  G(f  -  f) 


if 


;  TgG(f  -  f)  -  G(f  -  f);  ds  -  o. 


To  show  uniqueness,  suppose  v ^  and  are  two  solutions  of 

-A  v  =  f-f,  v(0)=0.  Let  W  =  vj-V2.  Then  Aw  =  0  and  since  w(0)  =  0  we 
must  have  AGw=0.  By  Dynkin's  formula 


G(n)w(x)  = E^fGCx^) w(xt) ]  for  all  t 


and  by  (4. 18)  it  follows  that 


lim  E^fGfx^)  w(xf)  ]  =  0  i.e.  w(x)  =  0  for  all  x. 
t  -*■  °0 


It  is  well  known  that  a  necessary  condition  that  Av=-f  have  a  solution 


is  that  n(f)  =  0.  See  Robin  [8], 


Theorem  4.4.  Under  the  assumption  (2.1),  (4.8),  (4.16)  and  (4.17)  a  necessary 
and  sufficient  condition  that 


(4.19)  Av  *  -  f 
have  a  solution  is  that  11(f)  =0. 

Moreover  if  fl(f)  =0,  then  any  two  solution  cf  (4.19)  differ  by  a  constant. 


Proof .  L'e  need  only  prove  the  sufficiency.  Suppose  Av  =  -f.  Then  by 


Theorem  4.1 


Av  =  -Gf  -  v(0) AG . 


Therefore  bv  Dvnkin's  formula 


v(0)  =  E  I  G(x  )  v(x  )  *  I  G (x  )  f  ( x  )ds+v(0)C(x  ) 
ol  t  t  J  s  s  tJ 


Letting  t 


(4.17) 


a 

v(0)  =E  I  I  G(x  )f(x  )ds  +  v(0), 
°  L  J  s  s  J 


that  is  n(f)  =  0 . 


Next  suppose  and  are  two  solutions  of 


Av  =  -  f . 


Then  by  Theorem  4 . 1 


A  G(v^  -  v2)  =  -(v^  -  v2) [0]  AG 


and  so  by  Dynkin's  formula  again 


G(x)(v1(x)  -  v2(x)) = ExfG(xt)(v1(xt)  -  v2(xt))] 


+  (vi-v2)(0)  El  AG(xs) 


U' 


Thus  by  (4.17) 


G(x)  (v  (x)  -  v  (x) )  =  lim  E  [G(x  )-G(x)](v 
1  2  t-*-«  X  1 


vL  (x)  -  v2  (x)  =  Vj  (0)  -  v2  (0) 


$1 


§5.  Asymptotics  of  a  Stopping  Problem. 


Assume  that 


(5.1)  f  ,\p  e  C  (R  )  with  f,ip^O 
b 


and  consider  the  stopping  problem 


(5.2)  u(x)  =  inf  J  (t) 

T 


where 


jx(t)  = 


•  -  If 

*r\ 


f  (x  )ds  +  (Kx  )  I  /  .  v 

S  T  ( T  <  00 ) 


Define  for  a  >  0 


J  (t)  =  E 

X  X 


l 


f  (x  )ds  +  e  aTiJj(x 


•>] 


and 


(5.3) 


,ct , 


u  (x)  =  inf  J  (t) . 
a  x 

T 


See  Bensoussan  [3]  or  Robin  [9]  for  the  following: 
Theorem  5.1.  Under  the  assumption  (2.1),  (2.2)  and  (5.1) 


(i) 


u  is  the  maximal  element  of  the  set  of  functions  h  e  C,  (R  ) 
a  ,t  b 


h^e  atTth  +  e  aSTgf  ds  h  ^ 


(ii) 


t  =  inf{t:u  (x  )  =  iHx  )}  is  optimal,  is 
a  at  t 


u  (x)  =  Ja(f  ) 
a  x  a 


(iii)  For  c  >  0,  define 


ja,C(U)  =  E 

X  X 


f°°  ft 

I  e  atexp  - j  l'gds  (f  (xt )  +  ^  V ^  \p  (x^ )  )dtj 

J  n 


where  0^1/  ^  1  and  1/  c  F  :  t  0 .  Let 
t  t  t 


w  (x)  =  inf  .1°  ’  C  <  U) 
a ,  e  y  x 


Ja. r-»V» ■ v  -  .V.-.V.V...V -VW O.V.V.V.VA, 


© 


r .  ■> 


V', 

VVS 


a 


«•> 


Then  v  is  the  unique  solution  in  C,  (R  )  of 

a,e  b 


w 


a ,  e 


i 


e  atT  [f-— (w  -^)  +  ]dt 


e  a,e 


and  moreover  if 


V  = 

t 


(  1  Ve^^V 

VO  w  (x  )  <  i|)(x  ) 


a,e  t 


then 


v  (x)=ja,c(U). 
a  ,e  x 


(iv)  w  I  v  as  e->-0  uniformly  on  compact  sets, 

a ,  £  a  -  r 


Suppose  we  have  the  additional  condition 
(5.4)  f(x)>e>0. 


Remark  5.1.  For  the  consideration  of  (5.1),  under  (5.3),  we  can  restrict  our 
attention  to  stopping  times  x  so  that 


Ex(T)<  f 


since  by  taking  x  =  0,  we  obtain  u(x)  ^  |  and  for  any  stopping  time 


x ,  3  E  [x]  <  J  (x)  . 

X  X 


For  the  following  see  Robin  [8],  Theorem  3.1  and  Remark  3.3: 


Theorem  5.2.  Under  (2.1),  (2.2),  (5.1)  and  (5.4) 


(i) 


u  is  the  maximal  element  of  the  set  of  funcetions 


h  £  C,  (R  )  h  <  T 
b 


,-f 

J  r\ 


T  fds  h  ^  ip 


(ii) 


x  =  inf{t:u(xt)  =  4*  ( xt ) }  is  optimal,  i.e. 


u(x)  =  -T  (x) 


(iii) 


u^i  u  uniformly  on  compact  sets  as  a  0 


( iv) 


Let  e  >  0  and 


w  (x)  =  inf  Jl(U) 


[V. . 


f.m.J.  fc  ».  m  a  r_»  T  M  f-a  ..A  ’lA  -  A.  tf-  Jt  -  A.  lEV  k!*^ 
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where 


J  (10  =  E 
x  x 


U®  t 

exp  £  J  ysdS)  ^f^xt)+  7  l/t'Kxt)dt 


and  0^1/  ^1,  1/  cF  :t^0.  Then  w  is  the  unique  solution  in  C,  (R  )  of 

s  t  t  e  b 


f 

w  =  T  w  +  i 

e  t  e  J 


T  (f  -  -  (W  -  Tj>)+) ds 
S  G  G 


and  if 


H 


1  w£(xt)  ^  M*,.) 


0  we(xt)  < ^(xt) 


w  (x)  =  J  (l/) 
e  x 


(v)  w  t  w  uniformly  on  compact  sets. 

Ot  j  G  G 


1  u  uniformly  on  compact  sets. 


Define  for  a > 0 


Jx(T)=E 


x[/  C(xt)f(y 


)dt  +  e  aXG(x 


)<Kx  ) 

t  T 


where  f  and  ip  satisfy  (5,1)  and  (5.4)  and  G(x)  is  a  continuous  distribution 
function  as  before.  Let 

Jx(t)  =  E^  G(x^) f  (x^)dt  +  G(xt)iMxt)^J. 

Under  the  assumption  (4.17),  it  is  consistant  to  define 

G(x  H(x  )  =  0  on  the  set  (x  =  °°) . 
x  x 

Hence  under  (4.16),  J  (x)  <  00  for  anY  stopping  time  x  and  moreover  by 
quasi-left  continuity,  see  (2.2) 

(5.5)  lim  J  ( x  A  t )  =  J  (x)  and  lim  Ja(x  A  t)  =  .Ja(x)  . 

X  X  _  v  x  X 

£  ->■  CO  £  ->  CD 


We  wish  to  establish  the  following  generalization  of  Theorem  5.2: 
Theorem  5.3.  Under  (2.1),  (2.2),  (4.16),  (4.17),  (5.1),  and  (5.4) 
(i)  u  is  the  maximal  element  of  the  set  of  functions 


heC,  (R  ),  h<T  h  +  I  T  (Gf)ds  h  <  . 

b  t  i  s 


T  =  inf{t:  u(xt)=G(xt)  >Kxt)}  is  optimal,  i.e. 


u (x)  =  J  (t) 


(iii)  + u  uniformly  on  compact  sets. 

The  proof  will  follow  after  a  series  of  lemmas.  Let 

J"(x)=Ex  j  f  (G(xt)V ^  f (xt)dt  +  G(xt)^(x^) j 


u  (x)  =  inf  J  (t) . 
n  x  x 


Lemma  5.4.  u  e  C.  (R  )  and  u  (x)lu(x). 
-  n  b  n 


Proof .  Note  that  by  (5.4)  and  Remark  5.1 


u  (x)  =  inf{J  (t):  E  [t]<°°}. 
n  x  x 


Since  for  any  x  with  E^[x]  < 


J  (t)  1  J  (t) 

X  X 

it  follows  that 

u  (x)  1  u(x)  u(x)^u(x). 

n 

Now  Theorem  5.4  apolies  to  u  (x)  and  so  for  anv  stopping  time  x 
‘  n 
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Letting  n-*-°° 


u  (x)  <  J  (t) 
n  x 


u(x)^Jx(t)  and  so  u(x)^u(x), 


i.e.  u(x)  =  u(x) . 


Let 

Je,n(l/)  =  E 


U°°  f  l/gds  -. 

e  1//e  0  (G(xt)V-^)  f  (xt>  +  ^VtG  (xt  )iKxt)dt  J 

0  .t 

r  r°  I  t/sds  1 

Jf  e  ate  1//e  0  G(xt)f(xt)+  -U/{.G  (xt)\J;(xt)dtJ 


Je*a(l/)=E  |  |  e~ate_1  /c*^ 

x  x 


and 


J  (10  =  E 
x  x 


o°  J"  Ugds 

|  e  ^ C  0  G(x  )f (x  )  +  G(x  )v|)(x  )dtl 
L  J  ttetttj 


where 


V  c  F  :t^0  with  0  <  \J  <1.  Define 
t  t  t 


u  (x)  =  inf  Je,n(l/),  u  (x)  =  inf  je*a(l/) ,  and  u  (x)  =  inf  J£ ( t/) 
e,n  ^  x  e,a  ^  x  e  ^  x 


Lenina  5.5. 


(i) 

u  eCL(R 

+> 

and  u  i 

e,n  b 

t  ,n 

(ii) 

u  (x)  e 

c  ,a 

Cb(E 

+  )  and  u| 

(iii) 

(x)  is 

the 

unique  non' 

u  =  T  u  +  f 
c  t  e  J 


e,a 


T  (Gf  -  I/E(u  -  G4/)  ) ds 
s  e 


V  = 

t 


1  uc(xt) ^ G(xt)^(xt) 
0  ue^xt)  <  G (x^ )i|i (x^ ) 


and  if 


Proof.  We  first  claim 


u  (x)  =  inf{J  (10 
e  x 


oo 

■f  V'- 


a .  s  .  P  } 
x 


OO 

(l  Vt < " ) 


Suppose  Px  I  I  l^dt  <  00  J  >0  and  let  5>0.  It  is  enough  to  show  that  there  is 


a  1/  :  t  ^  0  so  that 


00 

S’. 


dt  =  co  a.s.  P  and 
x 


jc (17)  <  Je(i0  +  5 

X  X 


By  (4.16) 


J  (10  <  E. 


x[/  G(xt)dt] 


(EfP  +  l/e|!tl! ) 


there  is  a  T  so  if  T>T  . 

o  o 


j  e~1/c  0  3  (G(xt)f(xt)  +  l/cVt  G(xt)i|»(xt)dt]  <6/2 


(5.6)  E. 


Not  that  (4.17)  implies  that 


lira  E  [G(x  )  ]  =0 

X  L 

t  OO 


and  so  there  is  a  so  that  if 


Ex[G(xT)]  <  6/2 M  • 


For  T^T  VT,  ,  define 
o  1 


1/  t  <  T 

t 

1  t  >  T 


-TOO  =  E 
x 


0 

dr 


-l/e-'O 


3 

G(xt)f(xt)  +  1  /cl/tG(xt)iJ;(x  )dt 


e'17^0  '  G(x 


Uxt)f(xt)  +  l/el/tG(xt)^(x  )dtj 


BUT. 


'CvVO-'VVV  'v.  -j\v  W*  Vo*-  •' 


Moreover  /  -  ^ 

E  r  j  e"1/e:*{)  S  G(xt)f(xt)dt 


)>>.[/ 


°  I  V  as 

e"1/cJ0  S  G  (x 


>(xt)f (xt)dt 


f  V  ds  _  r  /  f  V  ds- 

rj  e~l/e*'o  S  1/e  (x^_ )  dt  J  'A1'  je  C  0  s  G(xfc)  <  6/2 


Also  by  (5.6) 


^ t G  ( x ^  )  ij;  (x^  )  d  t  < 


and  thus 


J£(I/)  <  Je  («/)  +  6. 

X  X 


To  show  (i) ,  note  that  if  P 


CO 

XL  vt<“)> 


0  then 


(5.1)  and  (5.4)  imply 


f’-Wi  B/n  Ex  [  fe 

0 


Also  for  1/  =  1 


j“ ’  ( l/)  <  e/n  |'f|!  +  < 


-l/e-'o 


f  1/  d 
J  s  s 


dt  =  “ 


and  hence 


u  (x)  =  in£  {j£,n(l/):  i  1/  dt  =  »  a.s.  P  | 

c,n  x  It  x’ 

•'a 


Now  note  that  if  P 


l/^dt  =  oo  I  =  l  then 


(5.7)  J£,n(l/)  1  J  (U) 

x  x 


and  so  u  (x)  +  u  (x)  where  u  (x) ^  u  (x) . 
e ,n  e  e  e 


Since 


u  (x)^J£,n(l/) 
e  ,n  x 


\  A  A  .\Vo  A  A  A  A  A  A  '•  A 


and  for  v  :t^0  with  P 


V  dt  = 

t 


rl 


by  (5.7) 


we  have  u  (x)  =  u  (x) .  That  u  e  C,  (R  ) 
z  z  z ,n  b 

follows  since  Theorem  5.2  (iv)  applies. 

For  (ii)  first  note  that  J£((/)  <  00  for  all  :t^0 

x  t 


J£,a(l/)  +  J£(I/)  as  a-+-0.  Hence 
x  x 


u  (x)  1  U  (x)  u  (x)<u  (x) 
e  ,a  e  e  e 


By  taking  1/  =  1  we  see  that 

u  (x)^u  (x)<G(x)E  I  e  1//et(f(x  )  +  1/e  ^(x  ))dt 

£  £  X  I  t  t 


<  G(x)  (e|!  f  I'  + 


and  so 


(5>8)  lira  u£(x)  =  0. 


Bv  Theorem  5.1 


and  so 


u  (x)=eatTu  (x)  +  j  eaST(Gf-l/e(u  -GiJ;)+)(x)ds 
c,a  te,a  I  s  e,a 


u(x)=Tu.(x)+  j  T  (Gf  -  1  /  e(u  -G^)+)(x)ds 

C  t  r-  I  S  C,Ct 


For  any  V  : t  ^ 0  integrating  by  parts  yields 
/-t  _  ,-s 


r  v  ds  r*  .  .  [  v  dU 

:  *T)  S  u£(xt)+  I  e  eJ0  U  G 

r\ 


G (x  )f(x  )  +  1  /  e  H  u  (x  ) 
S  s  S  £  S 


-  1  /c  (u  -  G\l)+  (x  )ds 


is  a  martingale.  Taking 


1/  = 
t 


yields 


U£(x) 


0  ue(xt)  <  G(xt)iJ»(x  ) 


1  u£^xt^  ^  G  (xt  )(f<  (xc ) 


f  0  ds  rt 

■Exk1/e°  s  vv+[ 

Jn 


f  V  du 
•1/eJn  U  rt 


G(x  )f(x  )  +  l/cU  G(x  )^(x  )ds 
s  s  s  s  s 


and  by  (4.17)  and  (5.8)  it  follows  by  letting  t  -*■ <»  that 

u  (x)  =  J  (l')  and  so  u  (x)  =  u(x). 
ex  e 

UTiat  is  left  to  prove  in  (iii)  is  to  show  the  uniqueness, 
is  anv  non-negative  solution  of 


If  we  £  Cb(R  } 


[t 

=  T  w  +  ( 

C  £  J 

Jn 


T  (Gf  -  1/e(w  -  G^)+)ds 
s  e 


then  as  above  for  anv  (/  :t^0 

t 


(5.9)  w  (x)  =  E  e~ 1 1  ^0 

E  XL 


-1/e. 


Jo  (x  >+  [  -,,c 1 " 

Jr\ 


\J  du 


+  1  /  ct'gW  _  (xg )  -  l/c(wr  -  G^)  (xg)ds 


e  0  G ( x  )f(x  ) 

s  s 


Taking  1/  =  1  vields 


0^wg(x)^E_ 


and  letting  t -*•  =*■  we  obtain 


-t/e  ,  x  /  -s 
e  w  (x  )  +  |  e 
x  L  ct 

vn 


S^C  G(x  )f(x  )+l/eG(x  )\1(x  )ds 
s  s  s  s 


f 

•/o 


O^w  (x)  <  E  !  I  e  ^  G(x  )f(x  )+l/eG(x  )i Jj(x  )dt 
e  x  I  t  t  t  t 


G  ( x )  ( e  f  +  v  )  • 


.Vk>.  v'v. ■,  • 
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Thus  lim  w  (x)  =  0 . 
e 

x  -*■ 00 


$ 


From  (5.9)  it  follows  for  any  V 


w  (x)  <  E  e 
e  x 


j; 


V  ds 


[e-l/^0  's'*0..  N  ,  /  .-1/e, 


w  (xt)  +  j  e 


/ 


r  1/  du 

Jo  U- 


G(x  )f(x  )  +  l/el/  G(x  )0  (x  )ds 
s  s  s  s  s 


and  letting  t-*-°°  yields  by  (4.17) 


we(x)<J^(l/). 


Lastly  letting 

V  = 

t 

it  follows  that 


0  we(xt)  <  G(xt)i|j(xt) 


1  w£(xt)  >  G(xt)tJj(xt) 


w^(x)  =  J^(l/)  and  we  have  uniqueness. 


Remark  5.2.  Note  that  Lemmas  5.4  and  5.5  are  valid  for  the  case  when  G(x)4*(x)  is 
replaced  by 


(5.10)  if;  (x)  =  E 

P  X 


l 


e  ^SG(x^)»f;(x^)ds 


s  s 


since 


'l'g(x)  <  G(x)  . 


Lemma  5.6.  u  (x)  4u(x)  as  e -+•  0. 
-  e 


Proof .  An  argument  due  to  Menaldi  (see  [3]  or  [4])  shows  that  for  a >  0 


u  4-  as  e-*0. 

c,a 


Hence  Lemma  5.5  yields  u  4  as  e->0. 

e 


Define 


t  =inf{t:u  (x  )>i|/(x  )G(x  )}. 
e  e  t  st 


Lemma  5.5  implies  that 


u  (x)  =  E  fu  (x  )  +  I 
e  x  I  t  tAT  f 

Jn 


tAT 


G(x  )f(x  )ds 
s  s 


Since  on  t  =  00 
e 


uE(xt)  <G(xt)Mxt),  we  have 


lim  u  (x  )  =  0. 
t-.cc  e  tATe 

Because  of  quasi-left  continuity  cu  (t <  ") 


lim  u  (x  )  =  ^  (x  )>G(x  )^(x  ) 

_  .  m  e  t  At  e  t  t  t 

t  00  £  £  £  £ 


and  we  have 


u  (x)  ^  J  (t  )  ^  u  (x)  . 
e  x  e 


Let  (x)  be  as  in  (5.10).  Define  w  (x)  and  w(x)  by  substituting  \p. 

P  E  P 


in  place  of  G\p  in  the  definitions  of  u^_  and  u.  The  above  proof  again  yields 


w  ^  w . 

e 


Suppose  E^Ct)  <  00  ar*d  define 


0  t  <  T 


/T  = 


1  t  ^  T 


Then  for  w  and  w 
c 


Je(t/T 


)-J£(t)=E  ["  f  e  t\g(x  )f(x  )  +  1  /c  ip  „  (x  ))ds-t|/.(x)l 


40 


r-r. 


1 


$ 


v 

vy 

p: 


By  the  Markov  property  since  <p  e  D  ,  it  follows  that 

P  A 


'i'e<\  +  c,e 


-t/e 


-f 


e  S//£  <Atpg  -  1/e  ipg)  (xT  +  t)ds 


is  a  martingale  with  respect  to  =  + T ‘  Hence 


"WXT)] 


-'[l 

f 


s/z  /  *  i 


-1/£^)(XT  +  s)dS] 


e  S  T/,£  (Aip  -  1/eip  )(x  )ds 
P  p  s 


Hence 


J£((/T)  -  J 
x 


>W  =  E4 


e  S  T^C  G(x_) f  (x_)  -  AipQ (x_)ds 


V  '  s'  “rBv  s7 


<  e  ||  f  -  Aip 


6 


For  the  same  reasons  as  in  Lemma  5.4 


w(x)=inf{J  (t):E  [t]<°°} 
x  x 


and  so 


w^w  <  w  +  e|!  f  -  Aip  (I 
e  p 


showing  w^  1  w. 


Lastlv  since 


and 


that 


!|w  -  u!1  <  |!ip0  -  Gip|| 
p 


I'w  -  u  I1  <  !!  Ip  -  Gipl!  . 

t  t  p 


It  is  a  standard  fact  from  semi-grouo  theorv 


lim  I  ip  -  Gip  =  0  and  we  have  u  tu  as  e  -*■  0 . 
B  e 

a  -> 00 


Lemma  5.7.  u  +  u . 
-  a 


‘WyfiVj 


Proof .  Let  be  given  by 


where 


w  (x)  = inf  J  (t) 


j“(x)  =E  e  at  G(x  )f(x  )dt  +  e  aTi|i  (x  )  1 

X  XL  LL  pTJ 


and  w  (x)  be  defined  as  u  but  with  ip  replacing  G\p . 

C  ,  Ct  £  ,  Ct  p 

as  in  Lemma  5.6  that  ■  w  -w  !  <  ell  f  -  Aii,J  . 

e  ,a  a  B 

Hence  letting  w  and  w  be  as  in  Lemma  5.6, 

£ 


Then  it  follows 


|  W  (x)  -  w(x)  !  ^  !!  W  -•  W  f  +  |w  (x)  -  W  (x)  |  +  I!  w  —  w  |! 


a  a ,  £'  a ,  £ 


<  2  I'lp  -  &|»!|  +  ]w  (x)  -  w(x)  I 
a 

Letting  a 0  and  then  6  -*• «  shows  u  (x)->-u(x).  That  u  (x)  t  u(x)  follows 

a  a 

since  Ja(t)  t  J  (x) . 
x  x 

Proof  of  Theorem  5.3.  That  u  is  in  the  set  of  solution  in  (i)  follows  since 


Lemmas  5.4  and  5.7  show  ueC,  (R  )  and  letting  a  0  in  Theorem  5.1  (i)  vields 

b 


r 

u<  T  u+  I 

c  l 


T  (Gf)ds  u*SGi|>. 
s 


It  is  standard  that  if  h(x)  is  any  other  solution  then 


h(x)^Jx(x)  for  any  stopping  time  x.  Hence  u  is  the  maximal 


solution. 


What  remains  to  prove  is  that 


x  =  inf{t:u(xt)  =  GCx^HCx^)}  is  optimal, 


Let  B  =  {x:u(x)  =  C(x)G(x)}.  If  x £  B,  t  ;  0  and  there  is  nothing  to  prove. 
If  x  £  B,  find  £  so  that 

u  ( x )  +  6<t['(x)G(x). 


mmmm 


*  It*  I 


'Vi 

•wl 

/vi 


'M 


iV.il 

>y,*\ 

‘’••I 


I& 

wJ 

>  •»<! 


ii'i 


Define 


T  =  inf  {  t :  I  x  -  x  ^  R} 
R  1  t 


t  -inf{t:u(x  ) >  G(x  )<Kx  )  -  6} 
6  t  t  t 


Since  u  -*■  u  uniformly  on  compact  sets  choose  e  „  so  for  t  <  z  „ 

e  6  6 


sup  lu  (y)  -  u (y ) I  <  6/2. 
|x-y!  <R  £ 


For  s  e  [0,  A  xR) 


u  (x  )^u(x  )  +  6/2<G(x  )t|>(x  )  -  6/2 
c  s  s  s  s 


and  so  Lemma  5.5  savs 


•T«  Atr 


u  (x)  =  E 
e  x 


[».<Vatr>+  j  C<*,)f(VdS] 


Letting  e  -*•  0 


T6  ATR 

Ex[U<Xt,AxR)+  /  5<*s)f(*s)ds] 


Again  since  u=0  at  00  and  we  have  quasi-left  continuity,  we  can  let  R-*-" 


yielding 


i.ll)  u(x)  =  E  j"  u(x  )  +  f  G(x  )f(x  )ds  1 
xL  t.  J  s  sj 


Lastly  note  that  This  is  so  since  x^  t  t  as  6  ->-  0  and  by  quasi-left 


continuity 


x  -»•  x—  on  (  x  <  ®) 


T6  T 


Thus  u(x  ^  )  >  G(x  ~  )’p(x  ^  )  on  (  x  <  °°)  and  since  x^x,  x  =  x  on  (  t  <  °°) . 


Therefore  x  =  x  a.s.  P  .  Therefore  (5.11)  becomes 


J 

u(x)  =  E  I  u (x  -  )  +  I  G(x  )f(x  )ds  1  =  J  (x). 
x  l  x  ssj  x 


.?*■ 


The  Long  Run  Average  Cost  Problem. 

Let  Va(x)  be  as  in  Theorem  3.5.  Recall  that 


(6.1)  V  (x)  = inf 


nf  E  I  e  aSG(x  )f(x  ) +  a(Va(0) + c  )G(x  )ds 
x  x  L  J  ss  Os 

0 

+  e  aT  (Va(0)  +  g(xT)G(xT)+ cqG(xt)  )J 


Assume  (A. 8)  and  let 


r  (x)  = 


AG(x) 


Since  x  :  t  ^  0  and  G(x)  are  both  non-decreasing,  r(x)^0.  Define 


(6.1)  u  (x) 

a 


L 

x)  =  inf  E  j  e  asG(x  )(f(x  )  +  c  r(x  )-  aVa(0))ds 
t  <LJ  s  s  0  s 


+  e~aXG(x 


T)?(XT}] 


Bv  Dvnkin’s  formula 


=  E  [e  aTC(x  )  -  f 

x  L  x  J 


e  aS(AG(x  )-aG(x  ))ds 
s  s  -1 


(1  -  e  aT)Va(0)  =  a  e  aSds  Va(0) 


(6.3)  u  (x)  =  Va(x)+Va(0)  -  c.G(x) 

a  U 


and  if  x^  is  optimal  for  V  as  given  by  Theorem  3.5,  then  it  is  also  optimal 


for  u  (>:)  in  (6.2). 
n 

Define  E„ 


r  f  G(x  ) (f (x  ) + C„r(x  ) )dsl 


Note  that 


t 

[c<\>-  J 


AG  (  x  ^ )  d  s 


which  yields  by  (4.17)  by  letting  t  -*■  °° 


co  c: 

-of/  -«(xs)dSJ  -E0  f / 


G(x  )r(x  )ds 
s  s 


Hence 


(6.5)  V  = 


E0  J  G(*s>f(xs)dS  +c0 


Eq  J  G(xs)ds 


Lemma  6.1.  0  <  lim  aVa(0)^V. 


Proof.  Let 


oo 


e  aSG(x  )ds 
s 


where  a  is  given  by 


(2.3).  From  (6.1),  we  obtain  that 


0<Va(0)<E.  e  aSG(x  )f(x  )ds  +  (Va(0)  +  c.)H(a) 
0  J  s  s  J  0 


and  so 


:o  [  /  e" 


CKVa(0)  < - — 


°SG(x  )f(x  )ds J  + cn  H(a) 
s  s  J  0 


1  -  H (a) 


Since 


lim  1  -  H (a) 


’  00 

=  EQ(a)=E0  f  G(xg)ds  , 

L  •'a 


0  <  lim  a  V  '(0)  <  V. 


(x)  =  inf  J^J  e  G(x^)(f(x^)-V  (0))  +  ct  cn  G(x 


s>(f(xs)-V  (0)  )  +  a  cQ  G (x^ )ds  j 


+  e  (g(xT)G(xT)  +  cQ  G(x^) ) 


E  I  G  (x  )(f(x  )+c  r(x  )  -  V)ds 
x  J  s  s  Os 


v0(x)= 


Suppose  p  =  sup  Vq(x) 


(6.6)  V  +  p  r  ( x )  Q 


Theorem  6.1.  Under  (2.1)  -  (4.2),  (4.8),  (4.16),  (4.17)  and  (6.6) 


(i)  ->  v  uniformly  on  compact  sets  where 


t 

M  -  inf  [  J 


G(xg) (f (xg)  -  X)ds  +  g(xT)G(xT)  + CqG(x^) 


(ii)  t  =  inf{t:v(x^)  =  G(x^)g(xt)  +  CqG(x^) }  is  optimal, 

(iii)  v(x)  is  the  maximal  element  of  the  set  of  solutions  h  of  h  e  C,(R+) 

D 


[th+  J  Ts((E  "  ^)G)ds  h<Gg+cQG. 


Moreover  v(0)=0. 


(iv)  X »  Eq  J  G(xg) f (xg)ds +  G(xT)g(xT> +  cqG(xt) 


,[i 


Eq  J  G(xs)ds 


and 


\  =  inf 


E0  [  J  G(V  •  s 
0 


G (x  )f(x  ) d s  +  G ( x  ) g ( x  )  +  c  G(x  ) 

"  T  X  0  X 


Eo[  J5< 


E„  I  I  G(x  )ds 
s 


<  v  i 


•  <x  , 


1  in  aV  (O')  =  X 
\  0 


“ 0 


f(x  )ds+g(x  )I(x<o)  +  c  I(x>o) 

S  T  0 


Eq  [  X  A  o  ] 


I 


T  A  G 


f(x  )  d  s  +  g  ( :■:  )  I  (x  <  a)  +  c  I  (x  >  o) 
S  T  0 


mr 


Eq[t  a  cr] . 


Proof.  3y  Theorem  3.5  and  (6.3) 


v  (x)  =  Va(x)  -  Va(0)  =  u  (x)  +  c  G (x) 
a  a  0 


anci 


v ( x )  =  u(x) +  CqG (x) 


wnere 


u(x)  = inf  E  I  I  G(x  )(f(x  )  +  cnr (x)  -  A)ds  +  G(x  )g(x  ) 
TxIJs  sO  XT 


I.et 


e4/ 

-x 

[f 


E  I  J  G(x  )(f(x  )  +  c„r(x  )  -  V) )ds 
x  |  J  s  s  Us 


v0(x)  = 


G(x) 


and  v(x'  -  v(^(x)  -  p .  Note  that  v(x)  C  0  and  v(0)  =  -pGO.  De.fin 


w  (x)  =  u  (x)-G(x)v(x)  and 
a  x 


w(x)  =  u(x)  -  G (x)v (x) . 
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Bv  Theorems  4.1  and  4.4,  v(x)  is  the  unique  solution  of  “A  v  =  f (x)  +  c^r(x)  -V 


with  v (0 1  =  -  o  and 


-At  -  M£_x>  +  v(0)  ^ 


So 


— A(G  v) (x)  = G(x) (f (x)  +  c  r(x)  -  V) + v(0)AG(x) 

o 


Therefore 


-at- 


t 

e  ^""G(x  )v(x  )  +  (  e  “°G(x  )(f(x  )  +  cAr(x  )  -  V)  +  v(0)AG(x  )  -  aG(x  )v(x  )ds 
t  t  J  ssOs  s  ss 

0 


-as— . 


and 


t 

:V  +  / 


G(x  )v(x  )+  I  G  (x  )(f(x  )  +  c  r(x  )  -  V)  +  v(0)  AG(x  )ds 
ttjssus  s 


are  both  martingales.  Hence 

t 


l 

wq(x)  =  inf  j  e  aT  G(xs> (V  +  p  r(xc 


)  -  aV  (0))ds 


+  e  aTG(x) (g(x  )  -  v(x  )) 
X  T 


By  (4.17),  lim  G(x  )=0  and  by  (2.2),  x^it^O  is  quasi-left  continuous  and  so 
t  ->  °° 


for  all  stopping  times  x, 
T 


[/' 


E  I  I  G (x  )(f(x  )+cnr(x  )  -  X)ds  4-  G(x  )g(x  )  -  G(x)v(x) 

x|Js  sus  X  T 


[/*■ 


E  I  J  G(x  )  (V  +  p  r(x  )-X)+G(x  )(p(x  )  -  v(x  )) 
x  js  s  x  x  x 


and  so 


w(x)  = 


L 

=  int  E  I  G(x  )  (V  +  pr  (x  )-X)ds  +  G(x)(g(x  )  -  v(x  )) 
xxl/s  S  XT  X 

L  0 


Define 


w  (:■:)  =  inf  E 
a  x 


■  1/ 


e  aSG(x  )(V  +  pr(x  )-X)ds  +  o  aT  G (x  )  ( e  ( x  )  -  v ( %•  )) 
s  s  x  '  x  x 
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Let  a  -+  0  so  that  bv  Lemma  6.1, 
n 

lim  Van( 0)  =  X 

n 


Bv  Theorem  5 . 3 


w  (x)->w(x)  uniformly  on  compact  sets. 
an 


|  w  ( x )  —  w  (x)]<(|X-a  Van(0)  ]+  a  ||  v||)E  G(x  )ds 

n  n  11  n  n  '  v  c 


Since  (4 . 16)  sav 


E 

x 


G(x  )ds 
s 


<C 


independent  of  x 


lim  w  -  w  =  0 
»  a  a  ii 

n  -> 00  n  n 


Also 


jv^  (x)  -  v (x) |  ^  |wq  (x)  -  w(x) | 
n  n 


^  tw  -  w  +  w  (x)  -  w(x) 
a  a  1  a  1 

n 


and  so  (i.)  follows  for  the  sequence  {a^} 


From  Theorem  5.3 

x  =  inf{t:w(xt) -G(xt)(g(xt)  -  v^))} 

is  optimal.  It  follows  that 

x  =  in  f  { t :  v  (x  )  =  G(x  )g(xj  +  c  G(x  )} 
t  t  t  o  t 

is  optimal  for  v.  All  we  need  notice  is  that 

V(x)  -W(x)  +G(x)v(x)  +  C  G (x)  . 

o 

Since  v  (1)  =0,  it  follows  that  v(0)  =  0  and 
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=  inf 


E  G(x  )f(x  )ds  +  G  (x  )g(x  )  +  c  G(x  ) 

X  I  J  S  S  T  T  OT 


E0  J  G(xs)ds 


Thus  if  for  anv  other  subsequence  a  ■*  0  with  a  V  n(0)->X,  then  repeating 

n  n 

hne  above  argument  we  see  that  X  =  X.  Hence  (i)  and  (v)  follow.  Now  (ii) 
and  (iv)  are  also  done  and  (v)  follows  from  Lemmas  3.2  and  3.3. 

For  (iii)  ,  we  have  from  Theorem  5.4  that  w  is  the  maximal  solution  of 


It  C,  (R  )  £<t 

b 


.*♦  J 


Ts(G(V+pr-X))ds  £<G(g-v) 


Since 


=  TtG  v  -  J  Tg(G(f  +  cQr  -  V)  +  p  A  G)ds 


t 

G  =  TfcG  -  J  Ts(AG)ds 
0 


’  V  -  / 


T  (Gr)ds 
s 


we  see  that 


v(x)  =  w  ( x )  +  G  (x)  v(x)  +  cqG(x) 


_w  + 


<  T  w  +  T  (G(V  + pr  -  X) )ds 
t  s 


+  T  G  v  -  T  (G(f  +  c.r  -  V 
t  s  0 


+  c  T 
o 


tG  v  -J  V 

*0 

T  G  -  T 
o  t  J 


-  p  r ) )  d  s 


(G  r  rids 
s  0 


:  v  +  T  ( G ( f  —  a)  ids 
t  J  s 


Cn 


Gg  +  CqG  . 


Now  if  h  is  any  solution  of  (iv),  then 


h(x  )  +  G(x  )(f(x  )  -  X)ds  is  a  submartincal.e  and  so  for  anv 
t  s  s 


stopping  time  x, 


h ( x )  <E  h(x  .  )+  G(x  )(f(x  )  -  X)ds 

x  t  A  x  s  s 


G(xs)(f(xs)  -  X)ds  +  G(xtAT)  g(xtAT)  +  CoG(xt  At) 


By  (4.17)  and  (2.2)  letting  t  ->• °° 


h(x)  <  E  G(x  )(f(x  )-X)ds  +  G(x  )g(x  )+cG(x  )  . 

x  ss  x  xox 


is  optimal 


h  (x)  ^  v(x) . 


Remark  h . 1 .  Recall  that 

x  =  inf { t :  v(x£)  =  G(xt)g(xt> +  c0G(xt) } . 

1'r.der  the  assumption  g(0)  >0  since  v(0)  =0  it  follows  that 

PQ (•?  >  0)  =  1 . 

Remark  _h .  2 .  Since  Vq(0)  =  g  it  follows  from  Theorems  4.1  and  4.4  that  for  nn\ 

stomp inc  time  x. 


0  =  E  C ( x  )  vn  ( x  )  +  G (x  )(f(x  )  +  c.  r  (x  )  -  V)ds 
0  I  t  0  s  sos 


Theorem  6.2.  Suppose  for  UCR  open  and  not  containing  a  neighborhood  of 


that  if  x  =  i  nf  { t  :x(  t )  el’},  then  Pq(x<«>)>0.  Assume  further  that  G(x) 


for  all  x.  Under  Lho  conditions  (2.  1)  -  (2.4),  (4.8)  (4.  16),  (4.  17)  and 


a,  (0 1  >  0  ,  if  A  <  V  then  (g(x)<v(x)}^<J>.  If  r  (x )  >  8  >  0  and 


(g(x)  <  \-  .  <xl  }  ^  0  then  A  <  V. 


Proof.  Suppose  X < V  and  let  x  be  optimal.  Not  that 


(.6.71 


PqCt  <  “)  >  0. 


0  =  U-n  G(x  )(f(x  )+cnr(x  )  -  A)ds +  G(x-)g(x~) 
0  ssus  X  T 


;  cause  f  r  any  stooping  time  x, 


V«V  ’  Eo 


G(x  )r(x  )ds 
s  s 


'•'.'••"'irk  6.2  and  (6.8) 


0  =  h  G (x  ) (v  -  A)ds  +  G(x-) (g(x-)  -  v (x;) ) 

0  s  x  x  0  x 


v.vd  ov  Remark  6.1  and  (6.7) 


Pq(0  <  x  <  co)  >  0. 


g  ( :■: )  <  vQ  ( x )  }  t  <J>  • 


Su^n-vse  r(x)  >  ?,  >  0  and  {g(x)  <  v  (x) }  ^  Since  both  g  and  v^(x) 


■>nt  i  nu-  "ts  ,  for  f  >0  but  small  enough 


1  ±  rz(r.)  +  x  <  v  (x) }  and  does  not  contain  a  neighborhood  of  the 


■  i  e  i  n  .  .  >e ;  me 


,  r . ,  ■C,  vu_ c,  .* .  X  -  .  e. .  •  .  “  v 


:  *  ■•’.  •‘a 
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x  =  int'{t  :g(xt)  +  5  <  Vq(x^)  }  and  note  then  that  Pq(0  <  r  <  00 )  >0.  Also 

since  g(x)^0  and  (vQ(x)>g(x)}  4  $  then  p  =  sup  v^(x)  >  0  and  Theorem  6 . 1 
applies.  Arguing  as  above 


- 

T 

0^E0 

G(xg)  (V  -  A)ds  +  G(x^)  (g(x^)  -  vQ(x^.)) 

-J 

0 

and  so 


0  <  6 


Eo[G(\),<Eo 


G(x  )(V-  A)ds 
s 


that  is  A  <  V. 


Theorem  6.3. 

(i)  Under  the  conditions  (2. 1)— (2.4) ,  (4.8),  (4.16),  (4.17)  and  g(x)  > 0, 
if  A  =  V  and  r(x)^8>0  then  the  do  nothing  policy  is  optimal. 

(ii)  Under  the  conditions  (2 . 1 )— (2 .4) ,  (4.8),  (4.17)  and  if 
0  =  sup  vq(*)  =0  and  A  =  V  then  the  do  nothing  policy  is  optimal. 

Proof .  For  (i),  note  that  Theorem  6.1  applies  and  since 


G(x  )f(x  )ds 
s  s 


A  =  V 


-'0 


+  c 


G(x  )ds 
s 


the  do  nothing  policy  is  optimal. 

For  (ii)  if  p  =  supv^(x)  =0  we  have  for  any  stopping  time  t 


ft 


G(x  )(f(x  )-A)ds  +  G(x  )g(x  )  +  c  G(x  ) 

S  S  T  T  O  T 


rt 


G  (x  )  (V  -  A)ds  +  G(x  )  (g(x  )-v  (x  )) 
s  T  -  T  0  T 


•'0 


m 
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and  since 


C.(x  )(f(x  )  -  X)ds  +  c  =0 
s  s  o 


9 

wq 


again  the  do  nothing  policy  is  optimal. 
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